Skip to content

[xpu]support moe models on XPU platform #21643

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Aug 2, 2025
Merged

[xpu]support moe models on XPU platform #21643

merged 3 commits into from
Aug 2, 2025

Conversation

@yma11
Copy link
Contributor

@yma11 yma11 commented Jul 26, 2025 •
edited by github-actions bot
Loading

Essential Elements of an Effective PR Description Checklist

  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

Support models like ehristoforu/Falcon3-MoE-2x7B-Insruct, mistralai/Mixtral-8x7B-Instruct-v0.1 on XPU platform. For complicated models which require more moe releated kernels, will upstream support later as dependencies not published.

Test Plan

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn python3 examples/offline_inference/basic/generate.py --model ehristoforu/Falcon3-MoE-2x7B-Insruct --enforce-eager --dtype=float16 --trust_remote_code

Test Result

(Optional) Documentation Update

gemini-code-assist[bot]
gemini-code-assist bot reviewed Jul 26, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for Falcon3-MoE models on the XPU platform by adding XPU-specific logic to the FusedMoE layer. The changes leverage intel-extension-for-pytorch for the MoE implementation on XPU.

I've identified a couple of issues that should be addressed to ensure correctness and maintainability:

  1. A configuration parameter is hardcoded, which should be made configurable for consistency and performance tuning.
  2. The new forward_xpu method has an incomplete implementation that silently ignores crucial parameters. This is a critical issue that could lead to incorrect model behavior for other MoE models.

@github-actions
Copy link

github-actions bot commented Jul 26, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@jikunshang
Copy link
Collaborator

jikunshang commented Jul 28, 2025

can you rename the PR title? I think this change is not for falcon only. xpu should not support any MoE model before?

@yma11 yma11 changed the title support Falcon3-MoE on XPU platform support moe models on XPU platform Jul 29, 2025
@yma11 yma11 changed the title support moe models on XPU platform [xpu]support moe models on XPU platform Jul 29, 2025
@yma11
Copy link
Contributor Author

yma11 commented Jul 29, 2025

can you rename the PR title? I think this change is not for falcon only. xpu should not support any MoE model before?

updated. W/o this change, no MoE models can be supported but for models like deepseek, we need more kernels that are available in ipex 2.8.

@xuechendi
Copy link
Contributor

xuechendi commented Jul 29, 2025

@yma11 , with this PR, we will continue the optimizations on ipex.llm.modules.GatedMLPMOE right?
If this PR is ready, @jikunshang , please approve

@jikunshang jikunshang added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 29, 2025
@yma11 yma11 force-pushed the moe branch 2 times, most recently from 0597ffc to 753b8cd Compare July 31, 2025 07:06
jikunshang
jikunshang reviewed Aug 1, 2025
View reviewed changes
jikunshang
jikunshang reviewed Aug 1, 2025
View reviewed changes
@DarkLight1337
Copy link
Member

DarkLight1337 commented Aug 2, 2025

Can you merge from main to fix the CI failures?

yma11 added 3 commits August 2, 2025 11:38
@yma11
support Falcon3-MoE on XPU platform
3077879 
Signed-off-by: yan <yan.ma@intel.com>
@yma11
update
40bee1e 
Signed-off-by: Yan Ma <yan.ma@intel.com>
@yma11
fix comment
489937c 
Signed-off-by: Yan Ma <yan.ma@intel.com>
@vllm-bot vllm-bot merged commit 73e1b9b into vllm-project:main Aug 2, 2025
41 of 43 checks passed
npanpaliya pushed a commit to odh-on-pz/vllm-upstream that referenced this pull request Aug 6, 2025
@yma11 @npanpaliya
[xpu]support moe models on XPU platform (vllm-project#21643)
5a2149e 
Signed-off-by: yan <yan.ma@intel.com>
Signed-off-by: Yan Ma <yan.ma@intel.com>
jinzhen-lin pushed a commit to jinzhen-lin/vllm that referenced this pull request Aug 9, 2025
@yma11 @jinzhen-lin
[xpu]support moe models on XPU platform (vllm-project#21643)
a118857 
Signed-off-by: yan <yan.ma@intel.com>
Signed-off-by: Yan Ma <yan.ma@intel.com>
Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>
noamgat pushed a commit to noamgat/vllm that referenced this pull request Aug 9, 2025
@yma11 @noamgat
[xpu]support moe models on XPU platform (vllm-project#21643)
6fbd877 
Signed-off-by: yan <yan.ma@intel.com>
Signed-off-by: Yan Ma <yan.ma@intel.com>
Signed-off-by: Noam Gat <noamgat@gmail.com>
paulpak58 pushed a commit to paulpak58/vllm that referenced this pull request Aug 13, 2025
@yma11 @paulpak58
[xpu]support moe models on XPU platform (vllm-project#21643)
3f5fae3 
Signed-off-by: yan <yan.ma@intel.com>
Signed-off-by: Yan Ma <yan.ma@intel.com>
Signed-off-by: Paul Pak <paulpak58@gmail.com>
diegocastanibm pushed a commit to diegocastanibm/vllm that referenced this pull request Aug 15, 2025
@yma11 @diegocastanibm
[xpu]support moe models on XPU platform (vllm-project#21643)
112bf1c 
Signed-off-by: yan <yan.ma@intel.com>
Signed-off-by: Yan Ma <yan.ma@intel.com>
Signed-off-by: Diego-Castan <diego.castan@ibm.com>
epwalsh pushed a commit to epwalsh/vllm that referenced this pull request Aug 28, 2025
@yma11 @epwalsh
[xpu]support moe models on XPU platform (vllm-project#21643)
8d9162e 
Signed-off-by: yan <yan.ma@intel.com>
Signed-off-by: Yan Ma <yan.ma@intel.com>
zhewenl pushed a commit to zhewenl/vllm that referenced this pull request Aug 28, 2025
@yma11 @zhewenl
[xpu]support moe models on XPU platform (vllm-project#21643)
4290533 
Signed-off-by: yan <yan.ma@intel.com>
Signed-off-by: Yan Ma <yan.ma@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jikunshang jikunshang jikunshang approved these changes

+2 more reviewers

@xuechendi xuechendi xuechendi left review comments

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@yma11 @jikunshang @xuechendi @DarkLight1337 @vllm-bot